Skip to content

test(export): guard against memory exhaustion on large CSV exports - #1928

Merged
sublime247 merged 1 commit into
sublime247:mainfrom
uche001-dev:fix/csv-export-memory-exhaustion-1794
Aug 30, 2026
Merged

test(export): guard against memory exhaustion on large CSV exports#1928
sublime247 merged 1 commit into
sublime247:mainfrom
uche001-dev:fix/csv-export-memory-exhaustion-1794

Conversation

@uche001-dev

Copy link
Copy Markdown
Contributor

Closes #1794

Summary

Investigated src/routes/export.ts (the transaction bulk-export route) before writing any code, per this repo's own habit of checking whether described bugs still exist. It already uses pg-query-stream + Node's stream/promises pipeline() to stream rows from Postgres straight through a Transform into the HTTP response — no SELECT * result set is ever buffered fully in memory, in any of the CSV, JSON, or PDF export paths. There is no live memory-exhaustion bug in the current export route.

What this PR adds

A regression test that can actually catch a future reintroduction of buffering:

  • Streams 200,000 synthetic rows from a lazily-generated source (a generator function wrapped in Readable.from — deliberately never a real in-memory array of 200k rows, since building one would defeat the whole point of the test).
  • Asserts the HTTP response arrives as many small chunks, not one/a few large ones (maxChunkBytes < totalBytes / 5, dataEventCount > 10). A buffered implementation (collecting every row into an array or one big string before writing anything) would produce a single large data event close to the full response size — this assertion would fail against that regression and passes against the current, correctly-streamed implementation.

Why not assert on process.memoryUsage() heap deltas directly

I tried it first. GC timing under Jest/V8 makes heap measurements too noisy to assert on reliably — the same passing implementation intermittently failed a heap-growth bound by 2-3x from run to run with zero code changes. The chunk-shape assertion above is the deterministic, reliable signal for "streamed vs. buffered" and is what the test actually relies on.

Disclosed, not fixed here (pre-existing, unrelated)

tests/routes/export.test.ts — a second, older test file covering the same route with a different X-API-Key/ADMIN_API_KEY auth model — hangs indefinitely on every test in it. Confirmed present on a clean upstream/main checkout with zero code changes, reproduced both with and without a real local Postgres instance configured (ruled out "just needs a database" as the cause). Given the two files clearly test two different, non-overlapping generations of this route's auth model, this looks like a stale file left behind after a refactor rather than something in active use — flagging it rather than guessing at a fix, since I couldn't pin down the actual hang cause within a reasonable amount of investigation time.

Verification

  • npx jest src/routes/__tests__/export.test.ts — 7/7 passing (6 pre-existing + 1 new), run multiple times for stability.
  • npx eslint on the touched file — clean.

Testing requirements checklist (from the issue)

  • Implement/resolve changes associated with the memory-exhaustion concern — confirmed already resolved by the existing streaming architecture; added the missing regression coverage.
  • Ensure all existing and new unit tests compile and run successfully — src/routes/__tests__/export.test.ts (7/7). Note: tests/routes/export.test.ts is a separate, pre-existing broken file (see disclosure above), unaffected by and unrelated to this change.
  • Code passes lint.

Closes sublime247#1794

Investigated the transaction export route (src/routes/export.ts) before
writing anything: it already uses pg-query-stream + Node's stream/promises
pipeline() to stream rows from Postgres straight through a Transform into
the HTTP response, with no SELECT-then-buffer-then-send step anywhere in
the CSV, JSON, or PDF code paths. There is no live memory-exhaustion bug
in the current export route to fix.

What was missing was a test that could actually catch a regression back to
buffering. Adds one: streams 200,000 synthetic rows from a lazily-generated
source (a generator function wrapped in Readable.from, never a real
in-memory array of 200k rows, since that would defeat the point) through
the real route, and asserts the response arrives as many small HTTP chunks
rather than one/few large ones. A buffered implementation (e.g. collecting
every row into an array or a single string before writing) would produce
a single large data event close to the full response size; this test's
maxChunkBytes-vs-totalBytes assertion would fail against that regression
and passes against the current, correctly-streamed implementation.

Did not assert on process.memoryUsage() heap deltas directly - tried it,
but GC timing under Jest/V8 makes heap measurements too noisy to assert on
reliably (confirmed by hand: the same passing implementation intermittently
failed a heap-growth bound by 2-3x run to run with no code change). The
chunk-shape assertion is the reliable, deterministic signal for
"streamed vs. buffered" and is what this test relies on.

Also found and disclosing separately (not fixed here, pre-existing and
unrelated): tests/routes/export.test.ts (a second, older test file
covering the same route with a different X-API-Key auth model) hangs
indefinitely on every test, confirmed present on a clean upstream/main
checkout with no code changes at all, independent of database
availability (reproduced both with and without a real Postgres instance
configured).

Verification: npx jest src/routes/__tests__/export.test.ts - 7/7 passing
(6 pre-existing + 1 new), run multiple times for stability. npx eslint on
the touched file - clean.
@drips-wave

drips-wave Bot commented Aug 30, 2026

Copy link
Copy Markdown

@uche001-dev Great news! 🎉 Based on an automated assessment of this PR, the linked Wave issue(s) no longer count against your application limits.

You can now already apply to more issues while waiting for a review of this PR. Keep up the great work! 🚀

Learn more about application limits

@sublime247
sublime247 merged commit 1f43a76 into sublime247:main Aug 30, 2026
14 of 23 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] Resolve memory exhaustion during bulk CSV exports of transactions

2 participants